In three-dimensional image display devices, data representing a body contour (referred to as a surface model) in the form of three-dimensional position coordinates and normal vectors is stored in a storage unit, and using this data, an image of the body as viewed in any desired direction is displayed. Since the data stored in the storage unit is of the three-dimensional type, its volume when finely representing the body contour becomes large. Therefore, the quantity of calculation necessary in converting image data to obtain an image corresponding to any desired direction also becomes large. Conventionally, the direction in which the body is viewed is generally set by interactive control. In such an interactive control operation, numeric data representative of the direction is entered through an input unit such as a keyboard into a three-dimensional image display device. However, since the numeric data representative of the direction is less intuitive, it is very difficult for an operator to handle. To facilitate controlling, a realtime control system is also known. In such a system, a kind of hardware exclusive to conversion calculation is used to shorten the time of conversion calculation necessary to obtain an image as viewed in any desired direction, whereby the direction of a display image can be changed in realtime manner in linked relation to the operation of a coordinate input unit such as a track ball. However, such a device as including specific hardware is very expensive and poor in applicability.